Notes from Project meeting 10 September 2009 1000-1130 Bldg G Conference Room
Attending: Mike McCann, Francisco Chavez, John Ryan, [~reiko], Luke Beatman, Fred Bahr
2010 Data Visualization proposal feedback
In essence the feedback is:
For Phase 2 please address the following:
- Provide a status on the current state of One Stop Shopping project as well as projections of what will or will not be accomplished in 2009.
- Clarify what will specifically be developed, requiring the 1 FTE total for 2010.
- Clarify what additional capabilities this work will provide to the institute.
- Clarify why these addition capabilities are needed, who will be able to use them, and what data sets will be accessible for display with the new plotting/visualization capabilities.
We spent about 30 minutes discussing our response and came up with these items which will be included an additional 2+ pages to the written proposal:
- Focus on the CANON project (901009) which references using the infrastructure produced by the One-Stop Shopping and AOSN projects:
"To capture and integrate observed data in real-time, we will utilize infrastructure developments made as part of the "One-stop shopping proposal (901015) and collaborative tools developed under the AOSN proposal (900738)." - Emphasize that what we propose is linked with other efforts including CenCOOS, CANON, OceanSITES and that for the investment of 1 FTE software engineer we will also reap benefits from work provided by staff in these other groups.
- State clearly that the One-Stop Shopping project finishes with providing greater efficiency for the programmer, but not necessarily for the end user of the data
In specific responses to our average and below average scores:
| Criteria | Our response |
|---|---|
| Uniqueness: How unique is this contribution? | MBARI has a unique collection of platforms and instruments that are routinely deployed to measure upper ocean parameters. Providing access and visualization capabilities to the data sets produced from these deployments is transitively unique. |
| Timeliness: Why should this project move forward now? What are the drivers? | The One-Stop Shopping project finishes with providing greater efficiency for the programmer, but not necessarily the end user of the data. The 2010 Visualization project finishes what the 2009 One-Stop Shopping project started by implementing user friendly front-end to our data archives. With MBARI's implementation and operation of SSDS - producing richly described data sets - we have the opportunity to build upon that success and demonstrate and end-to-end ocean observatory data system with next year's CANON project. |
| Prior Productivity: Has the project leadership been successful with prior support on this project? | The project team members and leadership has been successful with previous MBARI data access portals, namely the Expedition Database and the Samples Database development and maintenance efforts. The team has also been responsible for the development and implementation of the Shore Side Data System for MOOS, OASIS, and AUVCTD data streams. |
| Does the cost benefit analysis show that this project is well justified? | We can perform an analysis of the various technologies and packages and implement the one (or two) that have the best cost/benefit ratio. In terms of evaluating the cost/benefit of the overall project we need to estimate the MBARI's expenditure in it's development and operation of all it upper water column measurement programs for an estimate of the cost term. In the case of projects such as CANON there is no data management component within the project - that project is depending on existing MBARI data management infrastructure (and this project) to make the data available in an integrated way. An additional cost is the expenditure for the proposed work in 2010 (1 FTE engineer and some funds for training). The benefit will be more efficient and interoperable access to data that are important to the success of several projects that depend on upper water column data. This is something that is difficult to assign a dollar value but maybe something easier to assign an subjective 'lack of frustration' value, or a quickness of data report preparation value. |
| Will the effort benefit a large number of users? | Yes. All of the research groups at MBARI that do work in the upper ocean will benefit as will CenCOOS and the other exteranal groups that we collaborate with such as OceanSITES, and the OOI. External users such as CenCOOS are already using the products developed by the One-Stop Shopping project and. We expect the relationship to continue next year as we add capabilities to the foundation developed in 2009. |
| How compelling are other justifications for this project? | There is really no other institution that combines an operational capacity and a mature observational data management system such as SSDS. We intend to leverage this capability to provide a unique data access and visualization system built upon community standards and conventions. |
Mike will work on revision to the to the proposal text an distribute a draft by tomorrow (Friday) evening. Edits to the draft need to be sent back to Mike by Monday morning 14 September 2009.
Data Access web page mock-up
Reiko showed her overall data access page which is a presented as a matrix with axes of platforms and parameters. The links in the cells should take a user to straight-forward pages that show how or provide access to the data (in many cells we have a lot of work to do).
THREDDS server status
SSDS managed HS2 data from M1 and M2 (2004 to the present) were added to the current catalog. This was done in response to Sergey's question whether the correct calibrations have been applied to all the deployments. We discovered that in fact the most recent deployments were not using the most recent calibrations and that oversight was fixed. Inter-comparison of the multiple deployments is now simplified by using TDS, e.g. here is a ferret script that plots the 3 HS2 variables from M1 and M2 over several deployments and still showing some issues with the data:
use "http://elvis.shore.mbari.org/thredds/dodsC/moorings/OS_M1_HS2" use "http://elvis.shore.mbari.org/thredds/dodsC/moorings/OS_M2_HS2" set reg/t=1-jul-2008:9-sep-2009 plot/vlimits=0:0.02 FL676[d=1],FL676[d=2] plot/vlimits=0:0.02 BB676[d=1],BB676[d=2] plot/vlimits=0:0.02 BB470[d=1],BB470[d=2]
Black lines are from M1, Red lines are from M2
The NetCheck program from Coastwatch program has been installed on pismo to periodically confirm that the data are accessible through the tomcat server on elvis. Email is currently being sent to Mike if there are any problems.
Web access portal for BOG data
Francisco asked for Reiko and Luke to think about this task. Mike discussed how MBARI could develop a web site similar to the Samples Database or Expedition Database query pages that not only provides access to the BOG relational database data, but also to the netCDF data (mooring, AUVCTD, Glider) that is available via OPeNDAP and TDS. Mike's preferred programming language is Python. It is a rapid development scripting language with powerful object orientated capabilities and a rich set of libraries available for relational database access and OPeNDAP data access (see the PyDAP example that access BOG bottle data from deeper than 4000 m) as well as output to various formats such as NetCDF, KML, simple text and custom graphics (see PyNGL PyNIO and matplotlib). A simple page providing access to BOG data can be developed in 2009 with an eye to extending the query to the netCDF data in 2010 should the Data Visualization project be approved.
RAMADDA
Unidata's RAMADDA server (just now in alpha release) was not examined. Instead we spent some time looking at how it would be to access and visualize upper water column data using Google Earth - as documented in the Setup for Google Earth dynamic queries miscellaneous document in this space.
901005_phase1_response.pdf (application/pdf)
bb470.gif (image/gif)
bb676.gif (image/gif)
fl676.gif (image/gif)
